Imaging apparatus and imaging system

ABSTRACT

An imaging apparatus of this application is an imaging apparatus for performing processing of machine learning related to estimation of a distance image closer to reality related to an object than a distance image related to the object captured by an imaging sensor from the distance image, including a data acquisition unit for acquiring distance image data related to the object; and a preprocessing unit for creating input data from the distance image data related to the object in which processing of machine learning for estimating distance image data close to reality related to the object from the distance image data related to the object is performed using the input data.

RELATED APPLICATIONS

The present application claims priority to Japanese Patent ApplicationNumber 2018-177328 filed Sep. 21, 2018 and Japanese Patent ApplicationNumber 2019-110661 filed Jun. 13, 2019, the disclosure of which arehereby incorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The application relates to an imaging apparatus and an imaging system,and particularly relates to an imaging apparatus and an imaging systemcapable of acquiring height information on an object.

2. Description of the Related Art

Conventionally, a system has been developed to detect athree-dimensional (3D) position, posture, and shape of an object. Forexample, there has been a known 3D vision sensor capable of capturing adistance image having distance information as a pixel value. Forexample, the 3D vision sensor acquires and analyzes images from twosensors (cameras) disposed to have parallax. The 3D vision sensorobtains a distance between a position corresponding to each pixel andthe sensor using a technology of triangulation using parallax. In thisway, the 3D vision sensor restores the 3D shape of the object (forexample, JP 2013-024653 A, etc.).

However, in a distance image captured by a 3D vision sensor of a stereosystem, etc., the shape of the object may be broken. For example, when aportion of an object whose distance to the 3D vision sensor rapidlychanges is acquired by the two sensors of the 3D vision sensor, it isdifficult to correctly associate portions projected on the respectiveimages with each other. For this reason, even in the case of pixelsindicating the portion of the object at positions whose distances to the3D vision sensor are equal to each other, different pieces of distanceinformation may be calculated between the pixels. In this case, theshape of the object in the distance image is broken.

Therefore, there is a desire for an imaging apparatus and an imagingsystem capable of estimating a distance image of an object closer toreality from a distance image of the object having a broken shape.

SUMMARY OF THE INVENTION

An aspect of the present disclosure restores a distance image in which ashape of an object is broken using machine learning according to thefollowing steps.

Step 1) Computer aided design (CAD) data of an object is acquired.

Step 2) A distance image of the object is acquired. A position/postureof the object on the distance image is acquired using a known analysismethod.

Step 3) The position/posture of the object on the distance image isconverted into a position/posture of the object viewed from a 3D visionsensor.

Step 4) CAD data of the object and the 3D vision sensor are disposed ina virtual space in a positional relationship between the 3D visionsensor and the object.

Step 5) A distance image close to reality related to the objectappearing in the 3D vision sensor in the virtual space is generated.

Step 6) A data set in which a distance image of a real world is inputdata and a distance image in the virtual space is label data is used aslearning data.

Step 7) Steps 1 to 6 are performed for the respective objects. Therespective objects may correspond to different types of objects.

Step 8) An acquired set of learning data is learned as a learning dataset by a learning device such as a neural network. When the distanceimage of the real world is input as input data, the learning deviceconverts the distance image into a distance image close to reality.

Further, an aspect of the present disclosure is an imaging apparatus forperforming processing of machine learning related to estimation of adistance image closer to reality related to an object than a distanceimage related to the object captured by an imaging sensor from thedistance image, including a data acquisition unit for acquiring distanceimage data related to the object, and a preprocessing unit for creatinginput data from the distance image data related to the object, in whichprocessing of machine learning for estimating distance image data closeto reality related to the object from the distance image data related tothe object is performed using the input data.

Another aspect of the present disclosure is a machine learningprocessing method of an imaging apparatus performing processing ofmachine learning for estimating a distance image closer to realityrelated to an object based on a distance image related to the objectcaptured by an imaging sensor, executing a first step of acquiringdistance image data related to the object, a second step of creatinginput data from the distance image data related to the object, and athird step of performing processing of machine learning for estimatingdistance image data close to reality related to the object from distanceimage data related to the object using the input data.

Another aspect of the present disclosure is an imaging system in which aplurality of apparatuses is connected to each other via a network, inwhich the plurality of apparatuses includes an imaging apparatusincluding at least a learning unit.

According to the aspects of the present disclosure, it is possible toestimate, from a distance image in which a shape of an object is broken,a distance image closer to reality related to the object than thedistance image.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-described and other object and characteristic of theapplication will be apparent from description of embodiments below withreference to accompanying drawings. In the drawings:

FIG. 1 is a schematic hardware configuration diagram illustrating animaging apparatus including a machine learning device according to anembodiment;

FIG. 2 is a schematic functional block diagram of an imaging apparatusaccording to a first embodiment;

FIG. 3 is a diagram illustrating distance image data and distance imagedata close to reality related to an object;

FIG. 4 is a schematic functional block diagram of an imaging apparatusaccording to a second embodiment;

FIG. 5 is a diagram illustrating an example of a three-tiered systemincluding a cloud server, a fog computer, and an edge computer;

FIG. 6 is a schematic hardware configuration diagram of an imagingapparatus mounted on a computer;

FIG. 7 is a schematic configuration diagram of an imaging systemaccording to a third embodiment;

FIG. 8 is a schematic configuration diagram of an imaging systemaccording to a fourth embodiment; and

FIG. 9 is a schematic configuration diagram of an imaging systemaccording to a fifth embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the application will be described withreference to drawings.

FIG. 1 is a schematic hardware configuration diagram illustrating animaging apparatus including a machine learning device according to anembodiment. The imaging apparatus 1 of the present embodiment is mountedon a computer such as a personal computer installed in a factory, a cellcomputer, a host computer, an edge computer, a cloud server, etc formanaging a machine installed in the factory. FIG. 1 illustrates anexample in which the imaging apparatus 1 is mounted on the personalcomputer installed in the factory.

A central processing unit (CPU) 11 included in the imaging apparatus 1according to the present embodiment is a processor that controls theimaging apparatus 1 as a whole. The CPU 11 reads a system program storedin a read only memory (ROM) 12 via a bus 20. The CPU 11 controls theentire imaging apparatus 1 in accordance with the system program.Temporary calculation data, various data input by an operator via aninput device 71, etc. are temporarily stored in a random access memory(RAM) 13.

For example, a non-volatile memory 14 includes a memory backed up by abattery (not illustrated), a solid state drive (SSD), etc. A storagestate of the non-volatile memory 14 is maintained even when the power ofthe imaging apparatus 1 is turned off. A setting area in which settinginformation related to an operation of the imaging apparatus 1 is storedis secured in the non-volatile memory 14. The non-volatile memory 14stores a program, data, etc. input from the input device 71 and CADdata, etc. read via an external storage device (not illustrated) or anetwork. A program and various data stored in the non-volatile memory 14may be loaded in the RAM 13 during execution/use. In addition, a knownanalysis program for analyzing a learning data set, a system program forcontrolling exchange with the machine learning device 100 describedbelow, etc. are written to the ROM 12 in advance.

For example, an imaging sensor 4 is a 3D vision sensor that generatesdistance image data (two-dimensional (2D) or one-dimensional (1D) arraydata having distance information as a value) of an object by capturingan image of the object. The 3D vision sensor as the imaging sensor 4 maycorrespond to a sensor of a triangulation system including a pluralityof charge coupled device (CCD) sensors, a sensor of a time-of-flightsystem, or a sensor of a focusing system. For example, the imagingsensor 4 is attached to a hand of a robot (not illustrated). The imagingsensor 4 is moved by the robot to an imaging position at which an imageof an object to be determined is captured. The imaging sensor 4transfers distance image data obtained by capturing an image of theobject to the CPU 11 via an interface 19. In addition, a distance imageof the object may be captured by fixedly installing the imaging sensor 4at a predetermined position and moving the object held by the hand ofthe robot to a position at which image capturing can be performed by theimaging sensor 4. A control operation related to image capturing of theobject by the imaging sensor 4 may be performed by the imaging apparatus1 executing a program. Imaging capturing of the object by the imagingsensor 4 may be performed under the control of a robot controller thatcontrols the robot or other devices.

Various data read into the memory and data obtained as a result ofexecution of a program, etc. are displayed on the display device 70. Inaddition, distance image data of an object obtained by image capturingby the imaging sensor 4, data output from the machine learning device100 described below, etc. are output via the interface 17 and displayedon the display device 70. In addition, the input device 71 including akeyboard, a pointing device, etc. receives an instruction, data, etc.based on an operation by the operator. The input device 71 transfers theinstruction, data, etc. to the CPU 11 via an interface 18.

An interface 21 connects the imaging apparatus 1 and the machinelearning device 100 to each other. The machine learning device 100includes a processor 101 that controls the entire machine learningdevice 100 and a ROM 102 that stores a system program, etc. The machinelearning device 100 includes a RAM 103 for temporarily storing eachprocess related to machine learning. The machine learning device 100includes a non-volatile memory 104 used for storing a learning model,etc. The machine learning device 100 observes each piece of information(for example, distance image data, CAD data, etc.) that can be acquiredby the imaging apparatus 1 via the interface 21. In addition, theimaging apparatus 1 acquires a processing result output from the machinelearning device 100 via the interface 21.

FIG. 2 is a schematic functional block diagram in a learning mode of theimaging apparatus 1 and the machine learning device 100 according to thefirst embodiment. Respective functions of functional blocks illustratedin FIG. 2 are realized by the CPU 11 of the imaging apparatus 1 and theprocessor 101 of the machine learning device 100 illustrated in FIG. 1executing system programs thereof and controlling operations ofrespective units of the imaging apparatus 1 and the machine learningdevice 100, respectively.

The imaging apparatus 1 of the present embodiment includes a dataacquisition unit 30, an object detection unit 32, and a preprocessingunit 34. The machine learning device 100 included in the imagingapparatus 1 includes a learning unit 110. CAD data related to an objectused for learning (a plurality of objects may be present) is stored in aCAD data storage unit 50 provided on the non-volatile memory 14 inadvance via an external storage device or a wired/wireless network (notillustrated).

The data acquisition unit 30 is functional means that acquires distanceimage data related to the object from the imaging sensor 4 or via anexternal storage device or a wired/wireless network (not illustrated).The distance image data acquired by the data acquisition unit 30 is datain which a distance from the imaging sensor 4 is set for each pixel of a2D imaging surface.

The object detection unit 32 is functional means that acquires, from thedistance image data related to the object acquired by the dataacquisition unit 30, a position and a posture of the object in thedistance image data. For example, the object detection unit 32 specifiesa shape of the object using CAD data related to the object read from theCAD data storage unit 50. The object detection unit 32 performs matchingprocessing between the CAD data and the distance image data whilechanging the position and the posture of the CAD data. In this way, theobject detection unit 32 specifies the position and the posture of theobject from the distance image data.

The preprocessing unit 34 creates teacher data used for learning by themachine learning device 100 based on the distance image data related tothe object and the CAD data related to the object. The preprocessingunit 34 creates teacher data T in which the distance image data relatedto the object is used as input data and distance image data close toreality related to the object generated from the CAD data (data of adistance image closer to reality than a distance image related to theobject acquired by the data acquisition unit 30 and data of an idealdistance image related to the object) is used as output date. FIG. 3 isa diagram illustrating an example of the distance image related to theobject and the distance image close to reality related to the object. Asdescribed above, in accordance with a distance image generationalgorithm, a distance detection error occurs in a portion of the objectin which the distance to the imaging sensor 4 changes. For this reason,the portion is easily broken. For example, the preprocessing unit 34disposes the CAD data at the same position and posture as the positionand posture of the object with respect to the imaging sensor 4 for aviewpoint position in virtual space, and generates distance image dataclose to reality related to the object from a shape of the CAD dataviewed from the viewpoint position at that time. In the distance imageclose to reality related to the object generated in this manner, noerror occurs even in a portion where a distance to a viewpoint changes.For this reason, the image is not broken. A jig, etc. for fixing theobject may be captured together with the distance image data related tothe object. In this case, the preprocessing unit 34 may acquire CAD datarelated to the jig in advance and dispose the CAD data related to thejig in the virtual space similarly to the CAD data related to the objectso that the CAD data is included in the distance image data close toreality related to the object.

The learning unit 110 is functional means that performs supervisedlearning using the teacher data T created by the preprocessing unit 34.The learning unit 110 generates (learns) a learned model used toestimate distance image data close to reality related to the object fromthe distance image data related to the object. For example, the learningunit 110 of the present embodiment may be configured to performsupervised learning using a neural network as a learning model. In thiscase, a neural network including three layers of an input layer, anintermediate layer, and an output layer may be used as the learningmodel. Alternatively, a neural network having three or more layers maybe used as the learning model. That is, a deep learning method may beused. In this case, more effective learning and inference are performed.The learned model generated by the learning unit 110 is stored in alearning model storage unit 130 provided on the non-volatile memory 104.The learned model is used for estimation processing of distance imagedata close to reality related to the object by an estimation unit 120.

The learning unit 110 repeatedly performs the above-mentioned learningusing distance image data related to various objects (different objects)acquired by the data acquisition unit 30 and distance image data relatedto the object captured by changing an imaging condition (an illuminationposition, etc.). In this way, the learning unit 110 generates a learnedmodel used to estimate distance image data close to reality related tothe object from the distance image data related to the object. Thegenerated learned model is used to acquire a distance image closer toreality regardless of the object or the imaging condition. That is, byusing the generated learned model, a distance image closer to reality isacquired regardless of how the object in the distance image obtainedfrom the imaging sensor 4 is broken.

In a modification of the imaging apparatus 1 of the present embodiment,the data acquisition unit 30 not only acquires distance image datarelated to the object, but also acquires image data of another formatsuch as luminance image data related to an object whose image iscaptured in the same positional relationship as a positionalrelationship between the imaging sensor 4 and the object at the time ofacquiring the distance image. The data acquisition unit 30 performslearning using the acquired image data of another format as auxiliaryimage data for assisting the distance image data. In this case, thepreprocessing unit 34 creates the teacher data T using the distanceimage data and the auxiliary image data related to the object as inputdata. The learning unit 110 performs supervised learning using thecreated teacher data T. In this way, a learned model for estimatingdistance image data close to reality related to the object isconstructed based on information related to more objects. By using thislearned model, accuracy of estimation of the distance image data closeto reality by the estimation unit 120 is improved.

In another modification of the imaging apparatus 1 of the presentembodiment, for example, the distance image data close to realityrelated to the object used when the teacher data T is created is createdbased on an image obtained using a high-precision imaging sensor.Examples of the high-accuracy imaging sensor include an imaging sensorhaving high resolution and a sensor that can detect luminance and otherphysical quantities in addition to the distance image and detect thedistance with higher accuracy based on the information. Even though suchan imaging sensor is expensive, the imaging sensor is used only at aconstruction stage of the learned model and can be used for otherpurposes after the learned model is constructed, which is an advantage.In addition, such an imaging sensor has an advantage that the imagingsensor can be used for construction of the learned model even when theCAD data of the object may not be obtained.

FIG. 4 is a schematic functional block diagram in an estimation mode ofthe imaging apparatus 1 and the machine learning device 100 according tothe second embodiment. Respective functions of functional blocksillustrated in FIG. 4 are realized by the CPU 11 of the imagingapparatus 1 and the processor 101 of the machine learning device 100illustrated in FIG. 1 executing system programs thereof and controllingoperations of respective units of the imaging apparatus 1 and themachine learning device 100, respectively.

The imaging apparatus 1 of the present embodiment estimates distanceimage data closer to reality than the distance image related to theobject based on the distance image data related to the object acquiredby the data acquisition unit 30 in the estimation mode. In the imagingapparatus 1 according to the present embodiment, functions of a dataacquisition unit 30 and an object detection unit 32 are the same asthose in the first embodiment.

A preprocessing unit 34 creates input data to be used for estimation inthe machine learning device 100 based on the distance image data relatedto the object.

An estimation unit 120 estimates an image close to reality related tothe object using a learned model stored in a learning model storage unit130 based on input data input from the preprocessing unit 34. Theestimation unit 120 of the present embodiment inputs input data(distance image data related to the object) input from the preprocessingunit 34 to a learned model generated (whose parameter has beendetermined) by supervised learning by the learning unit 110. In thisway, the estimation unit 120 estimates (calculates) distance image dataclose to reality related to the object. For example, the distance imagedata close to the reality related to the object estimated by theestimation unit 120 is displayed and output on the display device 70.The distance image data close to reality related to the object estimatedby the estimation unit 120 may be transmitted and output to a hostcomputer, a cloud computer, etc. via a wired/wireless network (notillustrated) and used.

In the imaging apparatus 1 of the present embodiment configured asdescribed above, distance image data close to reality related to theobject is estimated using a learned model obtained based on a pluralityof pieces of teacher data obtained by capturing images of variousobjects under various imaging conditions.

In third to fifth embodiments below, a description will be given ofembodiments in which the imaging apparatus 1 according to the firstembodiment is mounted as a part of a system interconnected with aplurality of devices including a cloud server, a host computer, a fogcomputer, and an edge computer (a robot controller, a controller, etc.)via a wired/wireless network. As illustrated in FIG. 5, in the third tofifth embodiments below, note that a system is configured by beinglogically divided into three layers of a layer including a cloud server6, etc., a layer including a fog computer 7, etc., and a layer includingan edge computer 8 (a robot controller, a controller, etc. included in acell 9), etc. in a state in which each of a plurality of devices isconnected to a network. In such a system, the imaging apparatus 1according to an aspect of the application may be mounted in any of thecloud server 6, the fog computer 7, and the edge computer 8. The imagingapparatus 1 may perform distributed learning by mutually sharing dataused in processing related to machine learning with each of theplurality of devices via a network. In addition, the imaging apparatus 1may collect the generated learning model in the fog computer 7 or thecloud server 6 to perform large-scale analysis. Furthermore, the imagingapparatus 1 may perform mutual reuse, etc. of the generated learningmodel. In the system illustrated in FIG. 5, a plurality of cells 9 isprovided in each local factory, and each of the cells 9 is managed bythe fog computer 7 in an upper layer in a predetermined unit (a unit ofa factory, a unit of a plurality of factories of the same manufacturer,etc.). Further, data collected and analyzed by the fog computer 7 isfurther collected by the cloud server 6 in an upper layer, and analysis,etc. is performed. Resultant information is utilized for control, etc.in each edge computer 8.

FIG. 6 is a schematic hardware configuration diagram in a case where animaging apparatus is mounted on a computer such as a cloud server or afog computer.

A CPU 311 included in an imaging apparatus 1′ mounted on a computeraccording to the present embodiment is a processor that controls theentire imaging apparatus 1′. The CPU 311 reads a system program storedin a ROM 312 via a bus 320. The CPU 311 controls the entire imagingapparatus 1′ in accordance with the system program. A RAM 313temporarily stores temporary calculation data or display data, variousdata input by the operator via an input unit (not illustrated), etc.

A non-volatile memory 314 is backed up by, for example, a battery (notillustrated). The non-volatile memory 314 is a memory, a storage stateof which is maintained even when the power of the imaging apparatus 1′is turned off. The non-volatile memory 314 stores a program input via aninput device 371 and various data acquired from respective components ofthe imaging apparatus 1′ or an imaging sensor 4, etc. via a network 5.The program and various data stored in the non-volatile memory 314 maybe loaded in the RAM 313 during execution/use. In addition, varioussystem programs such as a known analysis program, etc. (including asystem program for controlling exchange with a machine learning device100 described below) are written in the ROM 312 in advance.

The imaging apparatus 1′ is connected to the wired/wireless network 5via an interface 319. At least one imaging sensor, another imagingapparatus 1, an edge computer 8, a fog computer 7, a cloud server 6,etc. are connected to the network 5, and these devices mutually exchangedata with the imaging apparatus 1′.

Various data read into a memory, data obtained as a result of executionof a program, etc. are output to and displayed on a display device 370via an interface 317. In addition, the input device 371 including akeyboard, a pointing device, etc. transfers an instruction, data, etc.based on an operation by the operator to the CPU 311 via an interface318.

An interface 321 is an interface for connecting the imaging apparatus 1′and the machine learning device 100 to each other. The machine learningdevice 100 has a similar configuration to that described in FIG. 1.

As described above, when the imaging apparatus 1′ is mounted on acomputer such as a cloud server, a fog computer, etc., a function of theimaging apparatus 1′ is similar to that described in the first andsecond embodiments except that information from the imaging sensor 4 isexchanged via the network 5.

FIG. 7 is a schematic configuration diagram of an imaging systemaccording to a third embodiment including the imaging apparatus 1′. Animaging system 500 includes a plurality of imaging apparatuses 1 and 1′,a plurality of imaging sensors 4, and a network 5 that connects theimaging apparatuses 1 and 1′ and the imaging sensors 4 to each other.

In the imaging system 500, the imaging apparatus 1′ including themachine learning device 100 estimates a distance image close to realityrelated to the object whose image is captured by the imaging sensor 4using a learning result of the learning unit 110. In addition, at leastone imaging apparatus 1′ learns a distance image close to realityrelated to the object common to all the imaging apparatuses 1 and 1′based on the teacher data T obtained by each of the plurality of otherimaging apparatuses 1 and 1′. The imaging system 500 is configured suchthat the learning result is shared by all the imaging apparatuses 1 and1′. Therefore, in the imaging system 500, a more diverse data set(including the teacher data T) is used as an input, and a speed andreliability of learning are improved.

FIG. 8 is a schematic configuration diagram of a system according to afourth embodiment in which a machine learning device and an imagingapparatus are mounted on different devices. An imaging system 500′includes at least one machine learning device 100 mounted as a part of acomputer such as a cloud server, a host computer, a fog computer, etc.(FIG. 8 illustrates an example in which the machine learning device 100is mounted as a part of a fog computer 7), a plurality of imagingapparatuses 1″, and a network 5 that connects the imaging apparatuses 1″and the computer to each other. Similarly to hardware of the imagingapparatus 1′ illustrated in FIG. 6, hardware of the computer isconfigured by connection of hardware included in a general computer suchas the CPU 311, the RAM 313, the non-volatile memory 314, etc. via thebus 320.

In the imaging system 500′ having the above-mentioned configuration, themachine learning device 100 learns a correlation between a distanceimage of the object and a distance image close to reality related to theobject common to all the imaging apparatuses 1″ based on teacher data Tobtained for each of the plurality of imaging apparatuses 1″. In theimaging system 500′, a distance image close to reality can be estimatedfrom an image of the object captured by each imaging sensor 4 using alearning result thereof. According to the configuration of the imagingsystem 500′, a necessary number of imaging apparatuses 1″ can beconnected to the machine learning device 100 as necessary regardless ofan existing place or time.

FIG. 9 is a schematic configuration diagram of an imaging system 500″according to a fifth embodiment including the machine learning device100′ and the imaging apparatus 1. The imaging system 500″ includes atleast one machine learning device 100′ mounted on a computer such as anedge computer, a fog computer, a host computer, a cloud server, etc.(FIG. 9 illustrates an example in which the machine learning device 100′is mounted as a part of a fog computer 7), a plurality of imagingapparatuses 1, and a wired/wireless network 5 that connects the imagingapparatuses 1 and the computer to each other.

In the imaging system 500″ having the above-mentioned configuration, thefog computer 7 including the machine learning device 100′ acquires, fromeach of the imaging apparatuses 1, a learning model obtained as a resultof machine learning by the machine learning device 100 included in theimaging apparatus 1. Further, the machine learning device 100′ generatesa newly optimized or streamlined learning model by performing processingof optimization or streamlining of knowledge based on a plurality oflearning models. The machine learning device 100′ distributes thegenerated learning model to each of the imaging apparatuses 1.

Examples of optimization or streamlining of a learning model performedby the machine learning device 100′ include generation of a distillationmodel based on a plurality of learning models acquired from therespective imaging apparatuses 1. In this case, the machine learningdevice 100′ according to this example creates input data being input tothe learning model. The machine learning device 100′ performs learningfrom a beginning using an output obtained as a result of the input databeing input to each learning model. The machine learning device 100′generates a new learning model (distillation model) using such a method.As described above, the distillation model generated in this manner isdistributed to the imaging apparatus 1 or another computer via anexternal storage medium or the network 5 and utilized.

Next, another example of optimization or streamlining of a learningmodel performed by the machine learning device 100′ is shown. In aprocess in which distillation is performed on a plurality of learningmodels acquired from the respective imaging apparatuses 1, adistribution of output data of each learning model with respect to inputdata is analyzed by a general statistical method (for example, anoutlier test, etc.). An outlier of a set of the input data and theoutput data is extracted, and distillation is performed using the set ofthe input data and the output data from which the outlier is excluded.Through such a process, an exceptional estimation result is excludedfrom a set of input data and output data obtained from each learningmodel. As a result, a distillation model is generated using the set ofthe input data and the output data from which the exceptional estimationresult is excluded. The distillation model generated in this manner isutilized as a more versatile learning model when compared to learningmodels generated by the plurality of imaging apparatuses 1.

Other general learning model optimization or streamlining methods (suchas analyzing each learning model and optimizing a hyper parameter of thelearning model based on an analysis result) may be appropriatelyintroduced.

In the imaging system 500″ according to this example, for example, themachine learning device 100′ is disposed on the fog computer 7 installedfor the plurality of imaging apparatuses 1 as edge computers. In theimaging system 500″, learning models generated in the respective imagingapparatuses 1 are integrated and stored on the fog computer 7. In theimaging system 500″, optimization or streamlining based on a pluralityof stored learning models is performed, and an optimized or streamlinedlearning model is redistributed to each imaging apparatus 1 asnecessary.

In addition, in the imaging system 500″ according to this example, forexample, a learning model integrated and stored on the fog computer 7 ora learning model optimized or streamlined on the fog computer 7 may beintegrated on a host computer or a cloud server in a higher rank. Inthis case, application to intelligent operation in a factory or amanufacturer of the imaging apparatus 1 (construction and redistributionof a further general learning model in an upper server, support ofmaintenance operation based on an analysis result of a learning model,analysis of performance, etc. of each imaging apparatus 1, applicationto development of a new machine, etc.) is performed using these learningmodels.

As mentioned above, even though the embodiments of the application havebeen described, the application can be implemented in various modes byadding an appropriate change, without being limited only to the examplesof embodiments mentioned above.

For example, a learning algorithm executed by the machine learningdevice 100, an operation algorithm executed by the machine learningdevice 100, a control algorithm executed by the imaging apparatus 1,etc. are not limited to those described above, and various algorithmsare adopted.

In addition, even though the embodiments have described the imagingapparatus 1 and the machine learning device 100 as devices havingdifferent CPUs (processors), the machine learning device 100 may berealized by the CPU 11 included in the imaging apparatus 1 and thesystem program stored in the ROM 12.

1. An imaging apparatus for performing processing of machine learning related to estimation of a distance image closer to reality related to an object than a distance image related to the object captured by an imaging sensor from the distance image, the imaging apparatus comprising: a data acquisition unit for acquiring distance image data related to the object; and a preprocessing unit for creating input data from the distance image data related to the object, wherein processing of machine learning for estimating distance image data close to reality related to the object from the distance image data related to the object is performed using the input data.
 2. The imaging apparatus according to claim 1, wherein the preprocessing unit creates teacher data in which the distance image data related to the object is input data and the distance image data close to reality related to the object is output data, and the imaging apparatus further comprises a learning unit for performing supervised learning related to the processing of the machine learning based on the teacher data and generating a learned model for estimating the distance image data close to reality related to the object from the distance image data related to the object.
 3. The imaging apparatus according to claim 2, further comprising an object detection unit for detecting a position and a posture of the object from the distance image data related to the object acquired by the data acquisition unit.
 4. The imaging apparatus according to claim 2, further comprising a CAD data storage unit for storing CAD data related to the object, wherein the preprocessing unit generates the distance image data close to reality related to the object based on the CAD data related to the object stored in the CAD data storage unit.
 5. The imaging apparatus according to claim 2, further comprising: an object detection unit for detecting a position and a posture of the object from the distance image data related to the object acquired by the data acquisition unit; and a CAD data storage unit for storing CAD data related to the object, wherein the preprocessing unit generates distance image data close to reality related to the object based on the CAD data related to the object stored in the CAD data storage unit and the position and the posture of the object detected by the object detection unit.
 6. The imaging apparatus according to claim 2, wherein the preprocessing unit generates distance image data close to reality related to the object based on a distance image related to the object captured by an imaging sensor having higher accuracy than accuracy of the imaging sensor.
 7. The imaging apparatus according to claim 2, wherein the data acquisition unit acquires luminance image data of the object in addition to the distance image data related to the object, and the preprocessing unit uses the distance image data related to the object and the luminance image data as input data.
 8. The imaging apparatus according to claim 1, further comprising: a learning model storage unit storing a learned model for estimating the distance image data close to reality related to the object from the distance image data related to the object; and an estimation unit for estimating the distance image data close to reality related to the object from the distance image data related to the object using the input data and the learned model as the processing of the machine learning.
 9. A machine learning processing method of an imaging apparatus performing processing of machine learning for estimating a distance image closer to reality related to an object based on a distance image related to the object captured by an imaging sensor, the machine learning processing method executing: a first step of acquiring distance image data related to the object; a second step of creating input data from the distance image data related to the object; and a third step of performing processing of machine learning for estimating distance image data close to reality related to the object from distance image data related to the object using the input data.
 10. The machine learning processing method of the imaging apparatus according to claim 9, wherein the second step is a step of creating teacher data in which the distance image data related to the object is input data and the distance image data close to reality related to the object is output data, and the third step performs supervised learning based on the teacher data and generates a learned model for estimating the distance image data close to reality related to the object from the distance image data related to the object as the processing of the machine learning.
 11. The machine learning processing method of the imaging apparatus according to claim 9, wherein the third step estimates the distance image data close to reality related to the object from the distance image data related to the object using the input data and a learned model for estimating the distance image data close to reality related to the object from the distance image data related to the object as the processing of the machine learning.
 12. An imaging system in which a plurality of apparatuses is connected to each other via a network, wherein the plurality of apparatuses includes a first imaging apparatus corresponding to the imaging apparatus according to claim
 2. 13. The imaging system according to claim 12, wherein the plurality of apparatuses includes a computer including a machine learning device, the computer acquires a learning model as a result of learning of at least one of first imaging apparatuses, and the machine learning device included in the computer performs optimization or streamlining based on the acquired learning model.
 14. The imaging system according to claim 12, wherein the plurality of apparatuses includes a second imaging apparatus different from the first imaging apparatus, and a learning result by the first imaging apparatus is shared with the second imaging apparatus.
 15. The imaging system according to claim 12, wherein the plurality of apparatuses includes a second imaging apparatus different from the first imaging apparatus, and data observed in the second imaging apparatus is available for learning by the first imaging apparatus via the network. 